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ABSTRACT 

The value-added approach to the assessment of student 
outcomes examines actual or inferred changes in students' performance 
over time. It also attempts to separate the net affects of 
instruction from previous ability or simple maturation. Others may 
have different definitions of this concept, and this vagueness causes 
problems. Suggested ways to improve value-added assessment include 
the cross-sectional research design; methods of estimating the effect 
of a particular collegiate experience independent of students' 
pre-cellege differences; multiple regression analysis; analysis of 
joint or redundant effects not directly attributable to instruction; 
and the development of causal models. It is also noted that 
value-added assessments which consider only the general effects of 
college disregard the possibility that not all students may benefit 
equally from the same experience. Further attention to this area of 
research is needed. (GDC) 
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Are Value-Added 
Analyses Valuable? 



FINEST X I'ASCAKELLA 

Professor of Urban Etlucaiian Rmuirek Univmify of Illinois, Chicago 

One of the watchwords of higher education m the Nile 1970s and on 
into the igSas has been "accountability;' Public institutions 'in particular 
are being called upon b document their effects in terms of student 
learning and development (Fincher, igBdi Harlle, igB&l Often the 
impetus for this call to accountability is state government and the higher 
education coordinating or monitoring boards which, in a number of cases, 
report directly to the governor or the state legislature. In my own state, 
Illinois, I have had the opportunity, for the past year, to be a member of 
a committee chosen by the Illinois Board of Higher Education to study .he 
condition of undergraduate education in the state and make recommenda- 
tions to the state legislature. Two pertinent paragraphs from the final draft 
of the report recommendations are as follows: 

t, Golbges and universities shall conduct regular reviews of under* 
graduate education with emphasis on general education and the 
development of baccalaureate-level skills" The findings and conelu* 
sions of these reviews shall be reported to the Illinois Board of Higher 
Ed.. :ation* 

1, Each college and university shall assess individual student progress in 
meeting the objectives of general education and the development of 
haccalaureatedevel skills, and shall incorporate the results of assess- 
ment in the reviews of these areas, It is expected that colleges and 
universities will assess student progress at appropriate intervals and 
assure that assessment programs reinforce the maintenance of aca- 
demic standards, 

State-mandated student assessment in Illinois js certainly not an iso^ 
lated case, as an increasing number of states are mcving in a similar 
direction. Moreover, it is likely, based on the widespread impact of the 
recent National Institute of Education Report, Involvement in learning 
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Realizing the Potential of American Higher Education (Mortimer, el al„ 1984), 
that the current emphasis on estimating the student outcomes of higher 
education will be with us for the forseeable future, 

One of the more recognisable approaches to the assessment of student 
outcomes is the concept of "value-added" As I understand the term, 
value-added typically examines actual or inferred changes in students' 
performance over time. Students are assessed for entering com ? * mete 
on some set of reliable and valid instruments and then an ssed 
following a specified period of time (e.g., freshman to senior -. • or the 
completion of certain courses, programs of study, or other educational 
experiences (McMillan, 19B6). "Value-added" has another implication, 
however, which goes beyond simply looking at pre-post changes on 
fiomc measure of interest. Specifically, it attempts to separate that portion 
of student growth or development which can be reasonably attributed to 
specific educational experiences from that attributable to confounding 
causes such as differential ability or simple maturation, In short, value* 
added entaiU an estimation of the "net effects" of college, 

Of course this is my own understanding of what the term "value- 
added" implies. It must be acknowledged that its operational definition is 
likely fro vary with who uses the term, in what context and for what 
purpose (Eweil, 1086). For some Institutions, value-added my be inferred 
from simple freshman to senior changes in measures of cognitive develop- 
ment or learning (e^g„ Lenning, Munday and Maxey, 1969; Dumont and 
Troelstriip, 1981), For other institutions, the value of the education they 
provide may be evidenced largely by the accomplishments and retrospec- 
tive evaluations of graduates (e.g., Spaeth and Greeley, 2970; Pace, 1974)* 
Still other institutions may see value added largely as I have defined it, 
and may concentrate on estimating the net effects of the collegiate 
experience (e.g., Mentkowski and Stra^, 1983). Furthermore, even within 
the same institution, it can mean different things fro different administra- 
tive, faculty, student and alumni constituencies, Who decides what is 
valuable, for whom it is valuable, and to what it is added? 

Clearly value-added may not mean the same thing to all who use it. 
Indee* , as Ewell (tg&&) has pointed out, the operational definition of the 
term is still quite flexible and v igue, and this vagueness may vary in direct 
proportion to ih increasing use in public dialogue, 
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I. A Recent Deb 
Value-Added 



Perhaps in part bee ar operational definition of the 

term, the concept o icntly become the focal point of 

a spirited controvert- sl u >st respected names in the posisee* 

ondary scholarly corr Jiosen sides on the issue, Jonathan 

Warren (1934) has r enthusiastic and cogent entic..,m of 

value-added, The gin? Ar : ; >ment is that, while in the abstract the 

logic of value-added h i ir*itui appeal in practice it seldom leads any- 



where. The results of value-added analyses, he argues, "are often trivial 
difficult to make sense of and peripheral to most instructional purposes" 
(Warren, 1084, p, ml Warren's argument focuses largely on one level of 
analysis, the use of the value-added approach in assessing course-level 
instructional outcomes, Here he makes some telling points about the 
questionable practice of using pre-post differences as a measure of student 
learning in courses such as upper-division electromagnetic theory, where 
students can be assumed to have little pre^course content knowledge. 
Simply finding that students understand course concepts on the final 
examination is, he maintains, sufficient evidence to infer that most of the 
observed learning was a consequence of the course, A similar argument 
ha.s been made by Pace (1965), 

Warren's recommendation is that we need to abandon the unworkable 
concept of value- * Jded and get on with alternative ways of assessing the 
effects of postseec ndary education, Cameron Fincher (1985) is perhaps 
less convinced that value-added is a blind alley which needs to be 
abandoned, yet he is similarly skeptical in elucidating problems in its 
implementation, Thcte problems, he argues, center om 1^ *he develop- 
ment of reliable and valid instruments to measure various educational 
outcomes; 2) psychometric problems in assessing change; and ;,) concep- 
tual problems in interpreting college effects on achievement when most 
of the variation in achievement may be due to student aptitude, prior 
achievement and the quality of student learning effort rather than to 
instructional variables, While not necessarily ready to abandon the con- 
cept, Fincher is nonetheless skeptical about the ability of educators to 
apply value-added concepts to educational issues, He suggests that "if 
educators could agree on the assessable outcomes of higher education, 
take the time and effort to develop suitable forms of measurement and 
assessment, and restructure Instructional efforts in terms of explicit 
instructional objectives, value-added concepts of education might then be 
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a solution to same educational problems" (linchei, 1985, p, 398, author's 
emphasis). 

Pinchers points ire well taken. One must wonder, however, whether 
the issues which he elucidates are specific problems of n value-added 
approach to student as essment, A reasonable argument could be made 
that educational and behavioral research in general have traditionally 
been confronted with these and similar assessment issues, The knotty 
problems of instrument validity and attributing student learning to 
specific instructional practices are longstanding, if not adequately 
resolved, concerns of those interested in the effects of schooling at all 
levels (eg.. Wittrock, 1986), 

Responding directly to Warren s (1984) article, Astin (ig»$h) and 
Ewell have defended value-added as an important contribution to 

our thinking about assessing the impact of post secondary education, 
Doth authors readily admit that the approach is not without its problems, 
At the same time, however, they argue that it has conceptual strengths 
which outweigh these problems. These include: i) a focus on actual 
student development rather than typical measures of institutional "pres* 
tige" or "quality" (e.g., student body selectivity, resources per stedent, 
library size); a) the requirement of clearly defining, in conceptual jnd 
operational terms, the desired outcomes of college or other educational 
experiences; and 3) systematic attempts at assessing the impact of educa* 
tional experiences. They also argue, quite convincingly, that value-added 
as an approach to evaluating the impacts of college has been implemented 
In a systematic manner in only a few postsecondary institutions (e,g M 
Alverno College, University of Tennessee at Knoxville, Northeast Mis* 
sour! State University), Thus, abandoning value-added now would be the 
equivalent of dismissing an idea with considerable rational appeal "before 
it has been more extensively tried, evaluated and is better understood" 
(Astin, tg$$h f p. 1 t) f Clearly, Astin and Ewell want to avoid throwing the 
baby out with the bathwater— at least until we have a chance to see how 
the baby matures. 

Astin and Ewell make another point in their defense of value-added, 
Warren's critique, ihey suggest is based almost exclusively on a single 
level of analysis, i.e., student learning in a single course, Ewell concedes 
the point that there may be course situations involving a welUdefined, 
specialised body of knowledge to which students were not previously 
exposed where pretesting makes little or no sense, He further asserts that 
this is far from a valid critique of the entire value-added concept, however, 
since value-added is clearly useful in assessing cognitive and development 
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hi) outcomes of curricula and educational experiences more broadly 
contrived Hum individual counsel Indeed, it is in assessing these more 
broad-based effects that value-added has had its most typical application 
in postseeondary education. 

It seems unlikely that the current debate concerning the utility of 
value-added assessment is over, The term itself is to i value^ladcn and 
perhaps a bit overly pretentious in its public use by educators, In this 
sense it may appear to claim more than many current applications of 
value-added assessment have been able to deliver As a result it will 
probably continue to invite spirited criticism which, in turn, adds fuel to 
an ongoing debate, If this debate is to contribute light as Well as heat, 
however, it may be to our advantage to do two things, First, we might 
consider redefining value-added as the very fundamental and traditional 
research question; what are the student developmental outcomes associ- 
ated with exposure to an educational experience which can be reasonably 
attributed to the educational experience itself and not to other factors? 
This is the "net effects" question, and it is fir from being a new concern 
for educational researchers and evaluators (e.g., Campbell and Stanley, 
1963; Feldman and Newcomb, i960; Solmon and Tauhman, 19731 
Hymnn, Wright and Reed, [975; Bowen, 1977; Astia 1977, ig&zl 
Indeed a basic issue in educational research and evaluation has long been 
the extent to which we can attribute student development 1^ purposeful 
educational experiences, This is also, I believe, the core of what a 
value-added approach to student assessment is about. 

Given this perspective, a second thing we might do is to place the 
discourse about value-added on a different level, If we are willing to 
accept value-added as a potentially important approach to the assessment 
of student outcomes, then it behooves us to consider ways in which the 
methodology of the approach might be enhanced and sharpened, The 
remainder of this paper will suggest and discus.; a number of such 
methodological enhancements 

II. Pre- to Post^Changes: 

Improvement Can Be Misleading 

It is often the case that the value-added or net effects of an educational 
experience will be inferred from pre- to post-changes (say from freshman 
to senior year) on some accepted measure of student development (e.g., 
critical thinking, moral reasoning, reflective judgment act-comp scores), 



7 



75 



Unfortunately, even assuming the reasonable reliability of change scares, 
such mean changes reflect not only the effects of college, but also the 
effects of simple maturation, (Other variables such as history, instrument 
decay, and possibly even regression artifacts if the group is extremely low 
to begin with could also confound interpretation, but it is likely that 
maturation over time would be the major confounding variables) This, of 
course, means that longitudinal freshman-to-senior changes probably 
overestimate the effect due to college alone, U\, the unique effects of 
college (McMillan, 1986), 

One possible way to deal with maturation is through the use of a 
control grou r * of subjects followed over the same period of time, but not 
exposed to the particular educational experience (e.g., Plant, ig&i; 
Telford & Plant, 19631 Trent & Medsker, u?6&). In the situation where one 
is assessing institutional effects; however, reasonably comparable control 
groups not attending college are particularly difficult, if not impossible, to 
obtain. For such situations there are alternative cross-sectional or com- 
bined cross-sectional and longitudinal designs which provide reasonable 
controls for maturation. Consider the cross-sectional design where fresh- 
men are compared with seniors on a measure of critical-thinking ability, 
The freshmen, who have not been exposed to the institution, would act 
as a control group for the seniors who have theoretically benefited from 
four years of exposure to it, (To better reflect the entire college experi- 
ence, the measure of critical thinking might be given to freshmen upon 
enrollment in the institution anu to seniors in the final semester or quarter 
of their senior year,) The difference between the average freshman and 
average senior scores, statistically adjusted for differences in age, could be 
used to estimate the impact of the institution on critical thinking. 

There are, of course, potential problems with this design that must be 
addressed, First, seniors probably represent a more selective group in 
terms of academic ability since a portion of the least academically 
proficient are likely to have flunked out or to have left for academic 
reasons. Second, there is the possibility of differential recruitment or 
admission criteria being used for the seniors versus the current freshmen 
(e.g., if the institution used a more stringent set of admissions criteria for 
the current seniors as versus the current freshmen, the former might be a 
more academically select group than the latter). This might also lead to 
systematic group biases on a factor such as academic aptitude, which, in 
turn, is likely to influence the level of critical thinking. While acknowledg- 
ing that there is no ideal way to adjust for such pre-existing differences 
(Lord, 1967), one might nevertheless select the freshmen to be compared 
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from sat or act ranges similar to those of persisting seniors, and accom- 
pany this with regression analysis to provide aptitude- as well as age- 
adjusted critical-thinking means for the freshman and senior groups. The 
difference between the aptitude- and age-adjusted means would likely 
yield a better, though still imperfect, estimate of net institutional effects, 
than that yielded by simply adjusting for age alone, 

Cross-sectional designs such as this have recently been employed by a 
number of researchers in an attempt to separate college effects from those 
of maturation (e.g.. Whitla, 1978; Mentkowski and Strait, 1983), It would 
also seem reasonable thai such cross-sectional designs could be used in 
conjunction with pre^post longitudinal designs to provide a clearer 
picture of the influence of college versus the influence of maturation, 
Because the simple longitudinal results also include the possibility of 
maiurational influences, they might be thought of as the upper-bounds or 
liberal estimate of the effect of college. Conversely, the adjusted cross- 
sectional results would tend to statistically remove any joint impacts of 
the college experience and normal student maturation; thus providing a 
conservative or lower-bounds estimate of the effect of college The 
difference between the longitudinal and adjusted cross-sectional results 
might be used as an estimate of normal maturation during college, (In the 
absence of longitudinal data one might use the unadjusted differences 
between cross-sectional freshmen and senior cohorts to represent 
the upper-bounds estimate of college effects, the age and aptitude 
adjusted differences to represent the lower-bounds estimate, and the 
difference between the unadjusted and adjusted results to represent 
normal maturation,) 

Another cross-sectional design which has been used to disaggregate 
college effects from age or maturation effects is one which takes advan- 
tage of the increasing numbers of older, nontraditionaUage students in 
American postsecondary Institutions, In this design, traditionaUage fresh- 
men (e,g if age t$) t nontfaditionaUage freshmen (e T g„ age 22), traditional- 
age seniors (e.g age 22), and nontraditionaUage seniors (e,g„ age 26), 
migh* all be administered the measure of critical thinking, The effects of 
college versus maturation would be estimated by comparing the freshman 
to senior differences for both traditional and nontraditional students, w ^ 
age differences between traditional and nontraditionaUage freshmen and 
between traditional and nontraditionaUage seniors, Examples of this 
design have been used in estimating the effects of formal education on the 
development of reflective judgment (Perry, 1970) by Strange (1978), 
I offer these alternative designs not because they are without flaws, 
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Indeed, there are internal validity issues (i.e., what is the real cause of the 
effects observed) associated with both of them, As alternatives to simple 
pre« and post-assessments, however, they do provide somewhat greater 
control over the confounding influence of student maturation during 
college. Consequently, they probably provide more internally valid 
estimates of the value-added or net effects of educational experiences 
than do simple pre- to post-changes, In terms of contributing to our 
understanding of the value-added or net effects of educational experi- 
ences, pre- to posNchnnges in the absence of ,1 control group are quite 
limited, One might conceive of them as a necessary but insufficient 
condition for documenting educational impacts. In most instances, the 
presence of a net educational impact will be accompanied by positive pre- 
to post-changes on measures of interest, One must be a bit cautious in this 
regard, however. Recent evidence reported by Wolfle (7983), for exam- 
ple, has suggested that the general effect of college attendance on 
mathematics achievement is to maintain precollege levels of competence, 
Net of other factors, those not attending college tend to decline in 
mathemaHe competence. Thus a college or educational impact may not 
always be accompanied by pre- to posMmprovemcnt, or even by pre- to 
post-change. 

It is precisely in this type of situation that the term value-added can be 
misleading or even dysfunctional. As an estimate of college impact, 
value-added implies that something is added to the individuals level of 
development, In some areas of development, however, the impact of 
college (or other educational experiences) may be to prevent or retard 
decline rather than to induce major positive changes. Consequently 
approaches to value-added assessment which focus only on pre- to 
post-changes may be overlooking important college effects, 

III. Dir ect and Indirect Impacts 

I would argue that value-added assessment is made increasingly effective 
and useful in terms of its policy implications as it becomes more specific 
and focused. At the individual college level, this suggests the importance 
of identifying specific institutional experiences which enhance student 
development, Assessments for this purpose are often more fine-grained in 
conceptualisation and analysis than those which are limited to determin- 
ing the net effects of college or some particular educational experience. At 
the multi-institutional level we have some exemplary applications of this 
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approach in the work of Astin (eg., Astin, tg&& ( igyj, ig$2i Astin and 
Panes, tg&g) and Pace (1984)- Generally this approach involve* estimat- 
ing the associations between various measures of the college experience 
(e.g., academic major, academic achievement extracurricular involve* 
ment, Interaction with faculty) and certain outcomes (e.g., standardised 
measures of achievement, graduate degree attainment self-concept, val- 
ues) with the confounding influence of student pre-college traits (e.g., 
aptitude, secondary school achievement, social origins) removed statisti- 
cally, The result is an estimate of the effect of a particular collegiate 
experience independent of differences in stu lent pre-college traits. 

When multiple regression is employed to assess these partial associa- 
tions between college experiences and college outcomes, the resultant 
regression coefficients (either standardised or metric) can be interpreted 
as estimates of the net or direct influence of a particular variable (Wolfle, 
105), Thus, in addition to estimating the extent to which an institution 
facilitates the development of crih'cni-thinking ability from freshman to 
senior year, a value-added approach might also attempt to identify those 
particular Institutional experiences that have nontrivia! net associations 
with critical thinking, The results of these and similar analyses can provide 
useful information in terms of focusing attention on those potentially 
manipulable collegiate experiences that may causally influence the devel- 
opment of critical thinking (e,g„ particular courses or course-taking 
patterns, interaction with faculty), It is important to stress the "may" in 
the previous sentence, since attributions of causality with correlational 
data are tenuous at best 

Such analyses are an Important extension of value-added assessment 
and have been the analytic model for many benchmark studies of the 
influence of college on student development (see Feldman and Newcomb, 
1069; Bowen, 197?)* At the same time, however, these analyses are 
limited in their capacity to estimate the full range of a variable's influence, 
Because student pre-college characteristics and measures of the college 
experience typically ha\e substantial correlations between and among 
themselves, there is usually a substantial portion of the explained variance 
which cannot be uniquely attributed to any particular independent vari- 
able, This is typically referred to as the joint or redundant effects of 
student pre-college traits and the collegiate experience (Cohen & Cohen, 
1975). Figure 1 presents a schematic diagram showing the various unique 
(net) and joint or redundant effects, 

Joint or redundant effects have often gone unanalyzed, In a classic 
paper, however, Alwin and Hauser (1975) suggest that when independent 
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Figure 1; Schematic of College Effects 



Student Outcomes 
Of College 




a * Unique Effects of Student Pro-College Characteristics 

b » Joint Effects of Student Pre»Col(ego Gharnetarisiici Rnd the College Experience 
o » Unique Effects of College {Independent of Student Pfe-Colleflo Characteristics) 

variables are in a caudal sequence unanalyzed joint effects may be 
attributable to effects transmitted through intervening causes. Recent 
evidence, for example, has suggested that exposure to preenrollment 
freshmen orientation may have little direct influence on first-year persist 
tenee/withdrawal behavior, However, such orientation experiences may 
facilitate initial student social integration in college which, in turn, posi- 
tively influences persistence (Pasearella, Terenzint, & Wolfle, 1986), These 
are the indirect effects of a variable, and until comparatively recently they 
have been largely ignored in research on the impact of college on student 
development. With the increasing acceptance and use of causal model- 
ing as a research and analytical methodobgy, however, this need not 
continue. Causal modeling, which is essentially an attempt to fit a 
theoretical, explanatory model to a matrix of correlations among vari- 
ables, is the subject of a number of excellent discussions (e s g !# Anderson 
k Evans, 19741 Heise, 1075- Wolfle, igBm Maruyama & Walberg, 19821 
and Wolfle, 1985), 

Developing a causal model forces one to think theoretically, and, 
therefore, specifically and parsimoniously, One must specify not only the 
important variables (i.e., hypothesized causal influences) to be included in 
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the model but also the cr ma\ ordering and Ihu pattern of influences 
(CaUSal paths) nmong variables. These relationships are then cxprewd as 
structural equations, Structural equations, which are typically solved by 
regression analysis, specify how each variable (including the criterion) is 
a function of causally antecedent variables in the model, 

An important purpose of causal modeling is to portray the system of 
indirect as well as direct influences in a causal system, Consequently one 
is able to estimate not only the net direct influence of a variable on some 
outcome (i.e. the regression coefficient), but also the extent to which that 
variable influences intervening variables which, in turn, affect the out* 
come, The latter, of course, are the indirect effects, and are simply the sum 
of the products of direct effects through variables Intervening between 
the variable in question and the outcome measure. 

Because causal modeling permits one to portray the pattern** of indirect 
as well as direct effects on Home outcome, it yields a more complete 
estimation of the total effect (direct + indirect) of any particular variable, 
As such, causal modeling is a potentially important technique In value* 
added assessment, particularly if one is interested in understanding the 
process by which student development occurs rather than merely predict 
ing its occurrence. It is frequently the case that, net of other influences, a 
particular variable may have only a trivial or non-significant direct effect 
on student development, yet its indirect influence may be substantial and 
statistically significant; By their very niture traditional regression analy- 
ses that focus on prediction will overlook this indirect influence, and lead 
to conclusions that the variable has an unimportant influence on student 
development, 

Recent causal modeling analyses of the national Cooperative Institu- 
tional Research Project samples, for example, have suggested that college 
experience variables, mch as place of residence and the size and complex- 
ity of the Institution attended, have few if any net direct effects on student 
outcomes such as educational aspirations, academic self-concept, social 
self-concept, or humanitarian/civic involvement values, They do, how- 
ever, have significant indirect effects that are transmitted through their 
influence on level of student social and extracurricular involvement 
during college. Similarly, while academic achievement during college did 
not directly affect the subsequent occupational or economic attainments 
of Individuals/ It did have an important indirect effect on them by 
enhancing educational attainment (Pascarella, 1985b, 19850 Pascarella, 
Smart, Ethington and Nettles, ig$6i Pascarella, Smart U Stoecker, 1986). 

The point to be made here is not that value-added assessment should 
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he concerned with the particular variables in the above analyse**, Rather 
it is to suggest that the concern ot a\im\ modeling with understanding 
the patterns of direct iitid Indirect influences in a longitudinal process can 
provide a more complete and accurate estimation of the influence of 
specific educational experiences on student development, As such It is an 
important tool for sharpening the focus and increasing the understanding 
yielded by value -added analyses, 

IV, An Example of the Use of Caudal Models 
in Value-Added Assessment 

One area where causal modeling might significantly tmhanre value -<idd»'d 
assessment is in estimating the influence of differential eoursework and 
curricular patterns on cognitive development, It seems reasonable to 
assume that the nature of one's academic program will be a major, if not 
the major, influence on learning and cognitive development during 
college, Recent evidence from secondary school samples* for example, 
suggests that differential quantitative course work accounts for much of 
the gender difference in Scholastic Aptitude lest mathematics scores 
(Pallas and Alexander, 19S5), Thus, one might hypothesize that differeiv 
tiai pat terns of course work taken during college vvill have important 
direct effects on how much a student learns and m what areas he or she 
learns it (Pace, 1979), 

Beyond direct effects on learning, however, one might also he inter- 
ested in how a student's academic experience indirectly influences cogni* 
tive outcomes by Influencing the different dimensions of student Involve- 
ment in college, Astin (1984) defines student involvement as the extent 
or amount of physical and psychological energy that the student devotes 
to the academic experience, Involvement includes not only time spent 
studying and in laboratories, but also time spent interacting with faculty 
and participation in cultural, artistic and extracurricular activities which 
enl.oice the intellectual impact of the institution, Much if the research of 
Astin (e.g., Astin, 1977, tg$z t 19653) and others (e*g* t Pascarella, ig&o; 
Terenzini ik Pascarella, tg$o; Volkwcin, King & Terenzini, ig8§) has 
underscored the importance of student involvement as an influence on 
cognitive development, Apparently a good deal of what students learn 
during college is the direct result of their own efforts. These efforts* of 
course, are no doubt a function of individual student attributes. Net of 
these attributes, however, they are also likely to be formed, in part at leash 



82 



by the nature of the student's academic and coursework experiences. 
Methods of organizing or c|iiantlfyinf; student coursework or course* 
Liking patterns arc offered by Blackburn, Armstrong, Conrad, Didham & 
McKune (igy&h Prattler. Williams & VVadley ( toy oh lieeken [ty&zh 
McCombs k Smith (1986); and Ratcliff (1080), 

Recently Pace (1984) has developed an instrument (The College Stu- 
dent Experiences Questionnaire) that measures the student'* level of 
effort or involvement in various activities (e.g., studying, reading, attend- 
ing cultural events, interacting with faculty and peers) during college, 
The CSEQ is essentially a series of scales that estimate the amount, 
scope and quality of effort students invest in using the salient facilities 
and opportunities provided by the institution. As such, the CSEQ is a 
potentially important instrument for asM's*»ing Astin's { h;&i) concept of 
"involvement" 

Employing the CSEQ, one might develop an explanatory model of 
learning and cognitive development in college whic h posits that student 
coursework/curricular patterns are a function of student aptitude and 
various student background characteristics (e.g., social origins, educw 



Figure 2; A General Causal Model for A»»ea*iny the Effects of Student Coursework 
and Currteulsr Piftefnt on Student Learning and Cognitive Development 
During College 
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lional cind occupational aspirations, gender). In turn, net of aptitude and 
background characteristics, coursework and t utricular patterns would be 
expected to directly Influence extent of student involvement. Finally, 
measures of learning and cognitive development at the end of college 
would be seen as a function of entering academic aptileJe and student 
background characteristics, coursework and eurrlcular patterns, and 
extent of student involvement, Figure z in a graphic portrayal of the 
general causal model. 

Estimation of this model would permit determination of the total 
impact of differential coursework and cumeular patterns on measures of 
student learning and cognitive development at the end of college, One 
would be able to estimate not only the net direct effects, but also the 
indirect influence through student involvement, The indirect effects 
would Indicate the extent to which the institution's major structural 
mechanism for influencing student cognitive development (i.e., the aca- 
demic program) does so by influencing students' involvement in their 
own learning, Such evidence might suggest ways of structuring student 
coursework or cumeular pat terns to maximize both the direct and indirect 
effects on learning and cognitive growth, 

V, General Versus Conditional Effects 

This issue concerns the level at which value-added or net educational 
effects are assessed. Most existing attempts at value-added assessment at 
the institutional level have assumed that the impacts of educational 
experiences are general, that is, that the impact is essentially the same for 
all students. This assumption certainly has the appeal of parsimony (i.e., 
other things being equal the simplest explanation is often the optimal 
one). It can be argued, however, that assuming only general effects in 
one's analytic or assessment model ignores individual differences among 
students attending the same institution or exposed to the same educa- 
tional experiences to produce conditional rather than general effects. 
Thus, the magnitude of the influence of certain educa tional experiences on 
student development may vary for students with different characteristics 
(e,g„ level of entering aptitude, degree of prior exposure to, or compe- 
tence in specific course content, level of intellectual orientation), Condi* 
tional relationships such as this might well be overlooked in assessment 
approaches which consider only general effects. In certain situations this 
may lead one to conclude that effects of specific educational experiences 
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are trivial, when in fact they may have pronounced posit ve effect** for 
certain subgroups of students (Pascarella, 1985a). 

The notion of conditional effects determined by the interaction of 
individual differences among students with different methods of teaching 
or the presentation of course content is a respected tradition in Instruc- 
tional research, Here it is typically referred to as aptitude x treatment 
interaction (Cronbach & Snow, 7977), Underlying ite application in 
instructional research, however, is the mere genera! perspective, sup^ 
ported by the psychology of individual differences, that not all individu- 
als will benefit equally from the same educational experience, This idea 
may run somewhat counter to state-initiated mandates for accountability 
in which institutions are expected to demonstrate certain leveb of effec- 
tiveness in promoting cognitive and other development fur all students. 
Nevertheless, the consideration of conditional effects might well function 
to sharpen the focus of value-added assessment at the institutional level 
and enable it to better identify those particular students who are benefit- 
ing most or least from certain educational experienees i This information 
could then be used to focus institutional efforts on those student con- 
stituencies where its efforts appear to be least effective, Applications of 
the investigation of conditional effects in postsecondary education are 
shown in the work of Holland (1963) for career choice and academic 
achievement; Pfeifer (1976) for race and grades; Buenz and Merril (1968), 
Domino (1968), Pascarella (1978), ^oss & Rakow (1981), and Stinard & 
Dolphin (igBi) for different instructional approaches; Pascarella & Teren- 
zini ( 1979} and Bean (1985), in research on student attrition from college; 
and Pascarella, Smart, Ethington & Nettles {1986) in research on the 
development of self-concept during college, 



VI. Are Value-Added Analyses Valuable? 

My answer to this original question of the paper is a cautious M yes," 
Undoubtedly, a value-added type approach can make potentially impor- 
tant contributions to our knowledge about institutional impact. My 
cautions have to do with the term itself and with the need for its rigorous 
application at the institutional level if we are to provide policy makers 
with accurate and useful assessments of institutional impact As 1 have 
argued above, ViJue-added is perhaps current and overly pretentious 
terminology for a long-standing and basic issue in educational research; 
namely, what are the net effects of educational experiences on student 
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cognitive and non-cognitive development? It may he to our advantage in 
the future io remind ourselves of this, for in doing so we enn move more 
directly to substantive conceptual and methodological concerns 

Based on the above definition of the value-added approach, I have 
argued that .1 major concern should be on ways in which we can enhance 
the validity and usefulness of results yielded by value-added assessments, 
lb this end three methodological issues were discussed. First, it was 
argued that simple average change or improvement on some measure of 
student development often provides a misleading estimate of the long- 
term effects of college, Alternitive eross*scctlonal designs combined with 
statistical controls were suggested as ways to arrive at a more accurate 
estimate of the net effect of college. 

Second, it was* suggested that value-added analyses become increas- 
ingly useful in terrm of policy as they increase their focus on the specific 
aspects of the collegiate experience which may affect student develop* 
menL In addition to estimating the net effects of an institution, do they 
increase our understanding of how that institution functions to influence 
student development? To this end, causal modeling was suggested as an 
approach to value-added assessment which: 1) provided a theoretical 
template for understanding th e process by which the effects of college or 
other educational experiences occur, and a) permitted one to obtain a 
more complete estimate of the impact of various college experience 
mcasuref by estimating both direct and indirect effects on student 
outcomes, 

Third, It was suggested that value-added assessments that consider 
only the general effects of college disregard the very real possibility that 
not all students may benefit equally from the same educational experi- 
ences. Despite their usefulness in terms of a parsimonious estimate of 
educational effects, average differences or correlations may conceal as 
much as they reveal (Feldman & Newcomh, 1900), The more revealing 
question in terms of both policy and understanding of the complex 
dynamics of institutional impact is: what kinds of students change in what 
ways when exposed to what kinds of educational experiences? 

Finally, It should be pointed out that the accountability movement and 
its attendant concern for assessing the student development outcomes of 
postsecondary education is likely to be with us for some time. Ewell 
(1086) has suggested that no less than 16 states have, or are currently in 
the process of developing assessment plans for estimating the outcomes 
of public postsecondary education. This suggests that a good deal of 
postsecondary education policy, and perhaps even funding, will be based 
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on the? data we provide, If behooves us as educators and social si ien M sh 
to maximize the credibility of our assessments of student development by 
collecting data under the most rigorous conditions possible and analysing 
them in ways which increase our understanding the full range of specific 
institutional impacts, 
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